NAME¶

multipath.conf - multipath daemon configuration file

DESCRIPTION¶

multipath.conf is the configuration file for the multipath daemon. It is used to overwrite the built-in configuration table of multipathd. Any line whose first non-white-space character is a '#' is considered a comment line. Empty lines are ignored.

SYNTAX¶

The configuration file contains entries of the form:

<section> {

<attribute> <value>
...
<subsection> {

<attribute> <value>
...

}

}

Each section contains one or more attributes or subsections. The recognized keywords for attributes or subsections depend on the section in which they occor.

The following section keywords are recognized:

defaults

This section defines default values for attributes which are used whenever no values are given in the appropriate device or multipath sections.

blacklist

This section defines which devices should be excluded from the multipath topology discovery.

blacklist_exceptions

This section defines which devices should be included in the multipath topology discovery, despite being listed in the blacklist section.

multipaths

This section defines the multipath topologies. They are indexed by a World Wide Identifier(wwid), which is taken to be the value of the udev attribute given by the uid_attribute keyword.

devices

This section defines the device-specific settings.

defaults section¶

The defaults section recognizes the following keywords:

polling_interval

interval between two path checks in seconds. For properly functioning paths, the interval between checks will gradually increase to max_polling_interval; default is 5

max_polling_interval

maximal interval between two path checks in seconds; default is 4 * polling_interval

multipath_dir

directory where the dynamic shared objects are stored; default is system dependent, commonly /lib/multipath

find_multipaths

If set to yes , instead of trying to create a multipath device for every non-blacklisted path, multipath will only create a device if one of three condidions are met. 1 There are at least two non-blacklisted paths with the same wwid, 2 the user manually forces the creation, by specifying a device with the multipath command, or 3 a path has the same WWID as a multipath device that was previously created while find_multipaths was set (even if that multipath device doesn't currently exist). Whenever a multipath device is created with find_multipaths set, multipath will remeber the WWID of the device, so that it will automatically create the device again, as soon as it sees a path with that WWID. This should allow most users to have multipath automatically choose the correct paths to make into multipath devices, without having to edit the blacklist; Default is no

verbosity

default verbosity. Higher values increase the verbosity level. Valid levels are between 0 and 6; default is 2

reassign_maps

enable reassigning of device-mapper maps. With this option multipathd will remap existing device-mapper maps to always point to multipath device, not the underlying block devices. Possible values are yes and no. Default is yes

path_selector

The default path selector algorithm to use; they are offered by the kernel multipath target. There are three selector algorithms.

path_grouping_policy

The default path grouping policy to apply to unspecified multipaths. Possible values are

uid_attribute

The udev attribute providing a unique path identifier. Default value is ID_SERIAL

prio

The name of the path priority routine. The specified routine should return a numeric value specifying the relative priority of this path. Higher number have a higher priority. none is a valid value. Currently the following path priority routines are implemented:

prio_args

Arguments to pass to the prio function. This only applies to certain prioritizers

features

Specify any device-mapper features to be used. Syntax is num list where num is the number, between 0 and 6, of features in list. Possible values for the feature list are:

path_checker

The default method used to determine the paths state. Possible values are

failback

Tell multipathd how to manage path group failback.

rr_min_io

The number of IO to route to a path before switching to the next in the same path group. This is only for BIO based multipath. Default is 1000

rr_min_io_rq

The number of IO requests to route to a path before switching to the next in the same path group. This is only for request based multipath. Default is 1

rr_weight

If set to priorities the multipath configurator will assign path weights as "path prio * rr_min_io". Possible values are priorities or uniform. Default is uniform.

no_path_retry

Specify the number of retries until disable queueing, or fail for immediate failure (no queueing), queue for never stop queueing. Default is 0.

user_friendly_names

If set to yes , using the bindings file /etc/multipath/bindings to assign a persistent and unique alias to the multipath, in the form of mpath<n>. If set to no use the WWID as the alias. In either case this be will be overridden by any specific aliases in the multipaths section. Default is no

flush_on_last_del

If set to yes , multipathd will disable queueing when the last path to a device has been deleted. Default is no

max_fds

Specify the maximum number of file descriptors that can be opened by multipath and multipathd. This is equivalent to ulimit -n. A value of max will set this to the system limit from /proc/sys/fs/nr_open. If this is not set, the maximum number of open fds is taken from the calling process. It is usually 1024. To be safe, this should be set to the maximum number of paths plus 32, if that number is greated than 1024.

checker_timeout

Specify the timeout to use for path checkers and prioritizers that issue scsi commands with an explicit timeout, in seconds; default taken from /sys/block/sd<x>/device/timeout

fast_io_fail_tmo

Specify the number of seconds the scsi layer will wait after a problem has been detected on a FC remote port before failing IO to devices on that remote port. This should be smaller than dev_loss_tmo. Setting this to off will disable the timeout.

dev_loss_tmo

Specify the number of seconds the scsi layer will wait after a problem has been detected on a FC remote port before removing it from the system. This can be set to "infinity" which sets it to the max value of 2147483647 seconds, or 68 years. It will be automatically adjusted to the overall retry interval no_path_retry * polling_interval if a number of retries is given with no_path_retry and the overall retry interval is longer than the specified dev_loss_tmo value. The linux kernel will cap this value to 600 if fast_io_fail_tmo is not set.

queue_without_daemon

If set to no , when multipathd stops, queueing will be turned off for all devices. This is useful for devices that set no_path_retry. If a machine is shut down while all paths to a device are down, it is possible to hang waiting for IO to return from the device after multipathd has been stopped. Without multipathd running, access to the paths cannot be restored, and the kernel cannot be told to stop queueing IO. Setting queue_without_daemon to no , avoids this problem. Default is no

bindings_file

The full pathname of the binding file to be used when the user_friendly_names option is set. Defaults to /etc/multipath/bindings

wwids_file

The full pathname of the wwids file, which is used by multipath to keep track of the wwids for LUNs it has created multipath devices on in the past. Defaults to /etc/multipath/wwids

prkeys_file

The full pathname of the prkeys file, which is used by multipathd to keep track of the reservation key used for a specific WWID, when reservation_key is set to file. Defaults to /etc/multipath/prkeys

log_checker_err

If set to once , multipathd logs the first path checker error at logging level 2. Any later errors are logged at level 3 until the device is restored. If set to always , multipathd always logs the path checker error at logging level 2. Default is always

reservation_key

This is the service action reservation key used by mpathpersist. It must be set for all multipath devices using persistent reservations, and it must be the same as the RESERVATION KEY field of the PERSISTENT RESERVE OUT parameter list which contains an 8-byte value provided by the application client to the device server to identify the I_T nexus. If the --param-aptpl option is used when registering the key with mpathpersist, :aptpl must be appended to the end of the reservation key.

all_tg_pt

This must be set to yes to successfully use mpathpersist on arrays that automatically set and clear registration keys on all target ports from a host, instead of per target port per host. Default is no

retain_attached_hw_handler

If set to yes and the scsi layer has already attached a hardware_handler to the device, multipath will not force the device to use the hardware_handler specified by mutipath.conf. If the scsi layer has not attached a hardware handler, multipath will continue to use its configured hardware handler. Default is no

detect_prio

If set to yes , multipath will try to detect if the device supports ALUA. If so, the device will automatically use the alua prioritizer. If not, the prioritizer will be selected as usual. Default is no

detect_checker

If set to yes , multipath will try to detect if the device supports ALUA. If so, the device will automatically use the tur checker. If not, the prioritizer will be selected as ususal. Default is no

hw_str_match

If set to yes , the vendor, product, and revision parameters of user device configs will be string matched against the built-in device configs to determine if they should modify an existing config, or create a new one. If set to no , the user device configs will be regular expression matched against the built-in configs instead. Default is no

force_sync

If set to yes , multipathd will call the path checkers in sync mode only. This means that only one checker will run at a time. This is useful in the case where many multipathd checkers running in parallel causes significant CPU pressure. The Default is no

deferred_remove

If set to yes , multipathd will do a deferred remove instead of a regular remove when the last path device has been deleted. This means that if the multipath device is still in use, it will be freed when the last user closes it. If path is added to the multipath device before the last user closes it, the deferred remove will be canceled. Default is no

config_dir

If set to anything other than "", multipath will search this directory alphabetically for file ending in ".conf" and it will read configuration information from them, just as if it was in /etc/multipath.conf. config_dir must either be "" or a fully qualified directory name. Default is /etc/multipath/conf.d

delay_watch_checks

If set to a value greater than 0, multipathd will watch paths that have recently become valid for this many checks. If they fail again while they are being watched, when they next become valid, they will not be used until they have stayed up for delay_wait_checks checks. See "Shaky paths detection" below. Default is no

delay_wait_checks

If set to a value greater than 0, when a device that has recently come back online fails again within delay_watch_checks checks, the next time it comes back online, it will marked and delayed, and not used until it has passed delay_wait_checks checks. See "Shaky paths detection" below. Default is no

marginal_path_double_failed_time

One of the four parameters of supporting path check based on accounting IO error such as intermittent error. When a path failed event occurs twice in marginal_path_double_failed_time seconds due to an IO error and all the other three parameters are set, multipathd will fail the path and enqueue this path into a queue of which members are sent a couple of continuous direct reading asynchronous IOs at a fixed sample rate of 10HZ to start IO error accounting process. See "Shaky paths detection" below. Default is no

marginal_path_err_sample_time

One of the four parameters of supporting path check based on accounting IO error such as intermittent error. If it is set to a value no less than 120, when a path fail event occurs twice in marginal_path_double_failed_time second due to an IO error, multipathd will fail the path and enqueue this path into a queue of which members are sent a couple of continuous direct reading asynchronous IOs at a fixed sample rate of 10HZ to start the IO accounting process for the path will last for marginal_path_err_sample_time. If the rate of IO error on a particular path is greater than the marginal_path_err_rate_threshold, then the path will not reinstate for marginal_path_err_recheck_gap_time seconds unless there is only one active path. After marginal_path_err_recheck_gap_time expires, the path will be requeueed for rechecking. If checking result is good enough, the path will be reinstated. See "Shaky paths detection" below. Default is no

marginal_path_err_rate_threshold

The error rate threshold as a permillage (1/1000). One of the four parameters of supporting path check based on accounting IO error such as intermittent error. Refer to marginal_path_err_sample_time. If the rate of IO errors on a particular path is greater than this parameter, then the path will not reinstate for marginal_path_err_recheck_gap_time seconds unless there is only one active path. See "Shaky paths detection" below. Default is no

marginal_path_err_recheck_gap_time

One of the four parameters of supporting path check based on accounting IO error such as intermittent error. Refer to marginal_path_err_sample_time. If this parameter is set to a positive value, the failed path of which the IO error rate is larger than marginal_path_err_rate_threshold will be kept in failed state for marginal_path_err_recheck_gap_time seconds. When marginal_path_err_recheck_gap_time seconds expires, the path will be requeueed for checking. If checking result is good enough, the path will be reinstated, or else it will keep failed. See "Shaky paths detection" below. Default is no

missing_uev_wait_timeout

Controls how many seconds multipathd will wait, after a new multipath device is created, to receive a change event from udev for the device, before automatically enabling device reloads. Usually multipathd will delay reloads on a device until it receives a change uevent from the initial table load. The default is 30

skip_kpartx

If set to yes , kpartx will not automatically create partitions on the device. The default is no

ignore_new_boot_devs

If set to yes , multipath will never attempt to create a multipath device whose wwid is not listed in /etc/multipath/wwids, while running in the initramfs. This keeps multipath from adding new devices during the initramfs portion of bootup. The default is no

retrigger_tries

This sets how many times multipathd will reissue change uevents on block devices that are not blacklisted, but have no wwid set by udev. Multipath assumes that any devices that should not report a wwid are blacklisted. This means that if a non-blacklisted device has no wwid, it is likely that udev timed out while processing it. Multipathd will wait for a while, and then reissue a change uevent to give udev another chance to set the wwid. The default is 3

retrigger_delay

This sets how long multipathd should wait, after receiving a uevent for a non-blacklisted device without a wwid set by udev, before reissuing a change uevent. The goal of this delay is to give udev a chance to finish processing its current batch of uevents before sending more, to hopefully avoid it timing out. The default is 10

new_bindings_in_boot

If set to yes , multipath will allow new user_friendly_names bindings to be created while running in the initramfs. Otherwise, multipath will not create user_friendly_names bindings while running in the initramfs. Instead, it will use the WWID for the name of a device that was configured to use user_friendly_names. When multipathd is restarted later in boot on the regular filesystem, the device will be renamed to a user_friendly_name. The default is no

disable_changed_wwids

If set to yes, multipathd will check the path wwid on change events, and if it has changed from the wwid of the multipath device, multipathd will disable access to the path until the wwid changes back. The default is no

remove_retries

This sets how may times multipath will retry removing a device that is in-use. Between each attempt, multipath will sleep 1 second. The default is 0

max_sectors_kb

Sets the max_sectors_kb device parameter on all path devices and the multipath device to the specified value. Default is device dependent.

unpriv_sgio

If set to yes, multipath will set upriv_sgio on the multipath device and all its paths, when it is created or reloaded. The default is no

ghost_delay

Sets the number of seconds that multipath will wait after creating a device with only ghost paths before marking it ready for use in systemd. This gives the active paths time to appear before the multipath runs the hardware handler to switch the ghost paths to active ones. Setting this to 0 or off makes multipath immediately mark a device with only ghost paths as ready. The default is off

blacklist section¶

The blacklist section is used to exclude specific device from inclusion in the multipath topology. It is most commonly used to exclude local disks or LUNs for the array controller.

The following keywords are recognized:

wwid

The World Wide Identification of a device.

devnode

Regular expression of the device nodes to be excluded.

property

Regular expresion of the udev property to be excluded.

protocol

Regular expression of the protocol to be excluded. See below for a list of recognized protocols

device

Subsection for the device description. This subsection recognizes the vendor and product keywords. For a full description of these keywords please see the devices section description.

The protocol strings that multipath recognizes are scsi:fcp, scsi:spi, scsi:ssa, scsi:sbp, scsi:srp, scsi:iscsi, scsi:sas, scsi:adt, scsi:ata, scsi:unspec, ccw, cciss, nvme, and undef. The protocol that a path is using can be viewed by running multipathd show paths format "%d %P"

blacklist_exceptions section¶

The blacklist_exceptions section is used to revert the actions of the blacklist section, ie to include specific device in the multipath topology. This allows one to selectively include devices which would normally be excluded via the blacklist section.

The following keywords are recognized:

wwid

The World Wide Identification of a device.

property

Regular expresion of the udev property to be whitelisted.

protocol

Regular expression of the protocol to be whitelisted. See the blacklist section for a list of recognized protocols

devnode

Regular expression of the device nodes to be whitelisted.

device

Subsection for the device description. This subsection recognizes the vendor and product keywords. For a full description of these keywords please see the devices section description.

The property blacklist and whitelist handling is different from the usual handling in the sense that if the whitelist is set, it has to match, otherwise the device will be blacklisted. In these cases the message blacklisted, udev property missing will be displayed. For example settting the property blacklist_exception to (SCSI_IDENT_|ID_WWN) will blacklist all devices that have no udev property whose name regex matches either SCSI_IDENT_ or ID_WWN. This works to exclude most non-multipathable devices.

multipaths section¶

The only recognized attribute for the multipaths section is the multipath subsection.

The multipath subsection recognizes the following attributes:

wwid

Index of the container. Mandatory for this subsection.

alias

(Optional) symbolic name for the multipath map.

The following attributes are optional; if not set the default values are taken from the defaults or devices section:

devices section¶

The only recognized attribute for the devices section is the device subsection.

The device subsection recognizes the following attributes:

vendor

(Mandatory) Vendor identifier

product

(Mandatory) Product identifier

revision

(Optional) Revision identfier

product_blacklist

(Optional) Product strings to blacklist for this vendor

alias_prefix

(Optional) The user_friendly_names prefix to use for this device type, instead of the default "mpath"

hardware_handler

(Optional) The hardware handler to use for this device type. The following hardware handler are implemented:

The following attributes are optional; if not set the default values are taken from the defaults section:

Shaky paths detection¶

A common problem in SAN setups is the occurence of intermittent errors: a path is unreachable, then reachable again for a short time, disappears again, and so forth. This happens typically on unstable interconnects. It is undesirable to switch pathgroups unnecessarily on such frequent, unreliable events. multipathd supports two different methods for detecting this situation and dealing with it. Both methods share the same basic mode of operation: If a path is found to be "shaky" or "flipping", and appears to be in healthy status, it is not reinstated (put back to use) immediately. Instead, it is watched for some time, and only reinstated if the healthy state appears to be stable. The logic of determining "shaky" condition, as well as the logic when to reinstate, differs between the two methods.

"delay_checks" failure tracking

If a path fails again within a delay_watch_checks interval after a failure, don't reinstate it until it passes a delay_wait_checks interval in always good status. The intervals are measured in "ticks", i.e. the time between path checks by multipathd, which is variable and controlled by the polling_interval and max_polling_interval parameters.

"marginal_path" failure tracking

If a second failure event (good->bad transition) occurs within marginal_path_double_failed_time seconds after a failure, high-frequency monitoring is started for the affected path: I/O is sent at a rate of 10 per second. This is done for marginal_path_err_sample_time seconds. During this period, the path is not reinstated. If the rate of errors remains below marginal_path_err_rate_threshold during the monitoring period, the path is reinstated. Otherwise, it is kept in failed state for marginal_path_err_recheck_gap_time, and after that, it is monitored again. For this method, time intervals are measured in seconds.

See the documentation of the individual options above for details. It is strongly discouraged to use more than one of these methods for any given multipath map, because the two concurrent methods may interact in unpredictable ways.

KNOWN ISSUES¶

The usage of queue_if_no_path option can lead to D state processes being hung and not killable in situations where all the paths to the LUN go offline. It is advisable to use the no_path_retry option instead.

The use of queue_if_no_path or no_path_retry might lead to a deadlock if the dev_loss_tmo setting results in a device being removed while I/O is still queued. The multipath daemon will update the dev_loss_tmo setting accordingly to avoid this deadlock. Hence if both values are specified the order of precedence is no_path_retry, queue_if_no_path, dev_loss_tmo

SEE ALSO¶

udev(8), dmsetup(8) multipath(8) multipathd(8)

AUTHORS¶

multipath was developed by Christophe Varoqui, <christophe.varoqui@opensvc.com> and others.